Systems and Methods for Displaying Supplier Evaluation Data

ABSTRACT

Systems and methods for displaying supplier evaluation information include receiving a print order. A supplier data set is analyzed to determine a set of potential suppliers for the print order. For each respective supplier in the set of potential suppliers for the print order, one or more system-generated evaluation criteria values for the respective supplier are used to determine at least one representative system-generated evaluation criterion value for the respective supplier, and one or more user input evaluation criteria values for the respective supplier are used to determine at least one combined user input evaluation criterion value for the respective supplier. Supplier information for a subset of the set of potential suppliers for the print order is displayed, including identifying information, a representative system-generated evaluation criterion value, and a combined user input evaluation criterion value for a respective supplier from the subset of the set of potential suppliers.

RELATED APPLICATIONS

This application is a continuation application of and claims priority toU.S. patent application Ser. No. 15/368,529, filed Dec. 2, 2016,entitled, “Systems and Methods for Baselining using Multiple BaselineMethodologies,” which claims priority to U.S. Provisional PatentApplication No. 62/262,333, filed Dec. 2, 2015, entitled, “Systems andMethods for Baselining using Multiple Baseline Methodologies,” both ofwhich are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to print procurement systemsand, in particular, to generating supplier evaluation data usingmultiple types of evaluation criteria and displaying supplier evaluationdata in a print procurement system.

BACKGROUND

Print procurement systems allow suppliers of printing services toconnect with consumers of printing services. Some print procurementsystems allow the consumers (or an entity working on their behalf) toprovide detailed specifications related to a printing job. Thesedetailed specifications are then used by some print procurement systemsto match the printing job with a supplier that is capable of fulfillingthe printing job. In order to predict the cost of the print job,previous printing jobs with the same print specifications are identifiedand the price of individual items in a print job is extrapolated fromthe same types of items in previously fulfilled print jobs. However, insome cases types of items in a print job and/or numbers of items in aprint job vary from previously fulfilled print jobs.

SUMMARY

Without limiting the scope of the appended claims, after consideringthis disclosure, and particularly after considering the section entitled“Description of Embodiments” one will understand how the aspects ofvarious embodiments are implemented and used to address the abovedeficiencies and other problems associated with evaluating candidatesuppliers for a print job.

(A1) More specifically, some embodiments include a method of displayingsupplier evaluation values. In some embodiments, the method is performedat an electronic device with one or more processors and memory. Themethod includes: receiving a print order. In response to receiving theprint order, the electronic device analyzes a supplier data set todetermine a set of potential suppliers for the print order. For eachrespective supplier in the set of potential suppliers for the printorder, the electronic device determines, using one or moresystem-generated evaluation criteria values for the respective supplier,at least one representative system-generated evaluation criterion valuefor the respective supplier, and determines, using one or more userinput evaluation criteria values for the respective supplier, at leastone combined user input evaluation criterion value for the respectivesupplier. The electronic device displays supplier information for asubset of the set of potential suppliers for the print order. Displayedsupplier information includes identifying information, a representativesystem-generated evaluation criterion value, and a combined user inputevaluation criterion value for a respective supplier from the subset ofthe set of potential suppliers for the print order.

(A2) In some embodiments of the method of A1, the one or more inputevaluation criteria values are determined based on user input datareceived from a device that is remote from the electronic device.

(A3) In some embodiments of the method of A1, the one or moresystem-generated evaluation criteria values are automatically determinedbased on operations of an application.

(A4) In some embodiments of the method of A1, the at least onerepresentative system-generated evaluation criterion value for therespective supplier includes a timeliness value that is determined usingone or more quote response time duration values for the respectivesupplier and one or more invoice speed duration values for therespective supplier.

(A5) In some embodiments of the method of A1, the at least onerepresentative system-generated evaluation criterion value for therespective supplier includes a pricing and win rates score that isdetermined using a first ratio of winning quotes provided by therespective supplier to total quotes provided by the respective supplierand a second ratio of low bid quotes provided by the respective supplierto total quotes provided by the respective supplier.

(A6) In some embodiments of the method of A1, for each respectivesupplier in the set of potential suppliers for the print order, aranking for a respective supplier is determined using at least one of arepresentative system-generated evaluation criterion value for therespective supplier or a combined user input evaluation criterion valuefor the respective supplier.

(A7) In some embodiments of the method of A6, the subset of the set ofpotential suppliers for the print order includes a predefined number ofhighest ranked suppliers based on the determined rankings for eachrespective supplier in the set of potential suppliers for the printorder.

(A8) In some embodiments of the method of A6, the subset of the set ofpotential suppliers for the print order is displayed in an order that isbased on the determined rankings for each respective supplier in the setof potential suppliers for the print order.

In an additional aspect, an electronic device with a display, memory,and one or more processors is provided. The memory stores one or moreprograms which, when executed by the one or more processors, cause theelectronic device to receive a print order. The one or more programsfurther cause the electronic device to, in response to receiving theprint order, analyze a supplier data set to determine a set of potentialsuppliers for the print order. The one or more programs further causethe electronic device to, for each respective supplier in the set ofpotential suppliers for the print order, determine, using one or moresystem-generated evaluation criteria values for the respective supplier,at least one representative system-generated evaluation criterion valuefor the respective supplier, and determine, using one or more user inputevaluation criteria values for the respective supplier, at least onecombined user input evaluation criterion value for the respectivesupplier. The one or more programs further cause the electronic deviceto display supplier information for a subset of the set of potentialsuppliers for the print order. The displayed supplier informationincludes identifying information, a representative system-generatedevaluation criterion value, and a combined user input evaluationcriterion value for a respective supplier from the subset of the set ofpotential suppliers for the print order.

In another aspect, a non-transitory computer-readable storage medium isprovided. The non-transitory computer-readable storage medium storesexecutable instructions that, when executed by an electronic device,cause the electronic device to receive a print order. The executableinstructions further cause the electronic device to, in response toreceiving the print order, analyze a supplier data set to determine aset of potential suppliers for the print order. The executableinstructions further cause the electronic device to, for each respectivesupplier in the set of potential suppliers for the print order,determine, using one or more system-generated evaluation criteria valuesfor the respective supplier, at least one representativesystem-generated evaluation criterion value for the respective supplier,and determine, using one or more user input evaluation criteria valuesfor the respective supplier, at least one combined user input evaluationcriterion value for the respective supplier. The executable instructionsfurther cause the electronic device to display supplier information fora subset of the set of potential suppliers for the print order. Thedisplayed supplier information includes identifying information, arepresentative system-generated evaluation criterion value, and acombined user input evaluation criterion value for a respective supplierfrom the subset of the set of potential suppliers for the print order.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood in greater detail, amore particular description may be had by reference to the features ofvarious embodiments, some of which are illustrated in the appendeddrawings. The appended drawings, however, merely illustrate pertinentfeatures of the present disclosure and are therefore not to beconsidered limiting, for the description may admit to other effectivefeatures.

FIG. 1 illustrates an exemplary print procurement system, in accordancewith some embodiments.

FIG. 2 is a block diagram of a computing device, in accordance with someembodiments.

FIG. 3 is a block diagram of a print procurement server, in accordancewith some embodiments.

FIGS. 4A-4B are block diagrams illustrating data structures for storinginformation about print jobs, in accordance with some embodiments.

FIGS. 5A-5B are block diagrams illustrating data structures for storinginformation about baseline match tolerances, in accordance with someembodiments.

FIGS. 6A-6F illustrate user interfaces for displaying and interactingwith information about print jobs and for generating and displayingbaseline match candidates, in accordance with some embodiments.

FIGS. 7A-7B are a flowchart representation of a method of generatingbaseline candidates in a print procurement system, in accordance withsome embodiments.

FIGS. 8A-8P are schematics of a display used to illustrate exemplaryuser interfaces for a method of generating baseline candidates in aprint procurement systems, in accordance with some embodiments.

FIG. 9 illustrates a supplier evaluation interface, in accordance withsome embodiments.

FIG. 10 illustrates a supplier rankings interface, in accordance withsome embodiments.

FIGS. 11A-11B are a flowchart representation of a method of displayingsupplier evaluation values, in accordance with some embodiments.

In accordance with common practice, the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

DESCRIPTION OF EMBODIMENTS

Disclosed embodiments provide techniques for generating baselinescandidates using varying degrees of adherence between past and currentproject specifications, providing enhanced flexibility for identifyingbaseline candidates. Therefore, the disclosed embodiments improve printprocurement systems by facilitating generation of a greater number ofpotential baseline candidates.

FIG. 1 illustrates a print procurement system 100 in which someembodiments operate. One or more users each interact with one or moreuser computing devices 102 (also referred to as computing device 102 anddevice 102), such as a desktop computer, a laptop computer, a tabletcomputer, a tablet, a mobile computing device, a mobile phone, or avirtual machine running on such a device. In some embodiments, one ormore user devices 102 are used by a customer user (e.g., personnel froman entity that generates print orders). In some embodiments, one or moreuser devices 102 are used by a supplier user (e.g., personnel from anentity that bids on and/or fulfills print orders). In some embodiments,one or more user devices 102 are used by a facilitator user (e.g.,personnel from an entity that provides and/or maintains the softwareand/or hardware for receiving information (e.g., orders) from customersand receiving information (e.g., bids) from suppliers.

An example computing device 102-1 is described below with respect toFIG. 2, including various software programs or modules that execute onthe computing device 102-1. In some embodiments, the computing device102-1 includes a web browser 220 (FIG. 2) that is capable of executing aprint procurement web application 222 (FIG. 2), a print procurementapplication 224 (FIG. 2), a baseline candidate designation module 226, aprint job data fetching module 228 that the computing device 102-1 (oran application installed thereon such as print procurement application224) uses to generate baseline candidates. Some embodiments can generatebaseline candidates for use in a print procurement system withoutconnecting to external databases or programs over a network (i.e., thegenerating of baseline candidates is performed exclusively on the usercomputing device 102-1).

In some embodiments of the print procurement system 100, the usercomputing devices 102 communicate with the one or more print procurementservers 106 over one or more networks 104. The one or more networks(e.g., network(s) 104) communicably connect each component of the printprocurement system 100 with other components of the print procurementsystem 100. In some embodiments, the one or more networks 104 includepublic communication networks, private communication networks, or acombination of both public and private communication networks. Forexample, the one or more networks 104 can be any network (or combinationof networks) such as the Internet, other wide area networks (WAN), localarea networks (LAN), virtual private networks (VPN), metropolitan areanetworks (MAN), peer-to-peer networks, and/or ad-hoc connections.

An exemplary print procurement server 106-1 (also referred to herein asa server 106-1) is described in more detail with respect to FIG. 3. Asmentioned above, some embodiments provide a print procurement webapplication 222 (FIG. 2) that runs wholly or partially within a webbrowser 220 (FIG. 2) on the computing devices 102. In some embodiments,print procurement functionality is provided by both a local printprocurement application 224 and certain functions are provided by theone or more servers 106. For example, the one or more servers 106 may beused for resource-intensive operations.

In some embodiments, the computing devices 102 connect to one or moreremotely-located print procurement servers 106 through communicationnetwork(s) 104. In this way, some print procurement operations can beperformed remotely and computing resources at the computing devices 102can be preserved for other operations. For example, operations requiringinput from a user are performed at the computing devices 102 (e.g., theuser interacts with one or more user interfaces provided on thecomputing devices 102 (such as the user interfaces shown in FIGS. 6A-6Fand 8A-8P) and information identifying user input and user interactionsat the user interfaces are sent to the remotely-located printprocurement servers 106 for storage and/or further processing).

The one or more print procurement servers 106, in some embodiments,include a baseline candidate designation module 110 that designatesfulfilled print items as exact, similar, inferred, or no match baselinecandidates for a newly-created print item based on baseline matchtolerances. For example, print specifications for print items (i.e.,fulfilled and newly-created print items) can be sent to the one or moreprint procurement servers 106 through the communication network(s) 102for processing by the baseline candidate designation module 110 (i.e.,for identifying baseline candidates for each of the newly-created printitems). In some embodiments, sending and processing at least a portionof print items for processing at the one or more print procurementservers 106 preserves computing resources at the computing devices 102(i.e., because the servers 106 perform the data-intensive operations,such as scanning through large volumes of information in order toidentify baseline candidates, computing resources at the computingdevices 102 are preserved for rendering user interfaces and forsending/receiving data to/from the servers 106). In some embodiments,the allocation of operations between the computing devices 102 and theservers 106 is determined in accordance with bandwidth limitations atthe computing devices 102. For example, by monitoring a history ofbandwidth usage against available bandwidth at a respective computingdevice 102, a baseline measurement for bandwidth requirements at therespective computing device 102 can be established (e.g., if afterobserving/monitoring bandwidth usage for a predefined period of time(e.g., 2 days), bandwidth usage averages 100 MB per hour, then 100 MBhour can be established as the baseline).

By establishing the baseline, a determination can then be made todetermine whether to perform a data-intensive operation at the computingdevice 102 or at one of the servers 106. For example, in accordance witha determination that a first operation (e.g., identifying baselinecandidates) requires more than a predefined percentage (e.g., 10%, 20%,or 30%) of the baseline bandwidth, then the first operation should beperformed at one of the severs 106 and not at the computing device 102.In some embodiments, the baseline bandwidth is a baseline measuredagainst bandwidth usage over 20 minutes, 45 minutes, 1.5 hours, 6 hours,or the like.

In some embodiments, the determination as to which device should performa particular operation is based on where data required by the particularoperation is stored. For example, if the particular operation requiresdata that is stored in a respective print job database 108 (FIG. 1),then the particular operation can be performed more efficiently at oneof the servers 106, because the required data can bequeried/retrieved/processed locally and without having to sending thatdata back and forth using the networks 104, in order to communicate thatdata to one of the computing devices 102.

In some embodiments, the information about fulfilled print items (e.g.,print specifications for each of the fulfilled print items, purchasinginformation, supplier information, etc.), newly-created print items(e.g., print specifications for each of the fulfilled print items,associated baseline match tolerances for the newly-created items), andbaseline candidates are all stored in the one or more print jobdatabases 108 for easy access and storage.

In some embodiments, the print procurement system 100 is provided by aprint procurement service provider. The print procurement serviceprovider connects entities requiring print jobs to create printedmaterials (e.g., banners, posters, business cards, etc.) with supplierscapable of fulfilling the print jobs. In some embodiments, print jobseach include one or more items (i.e., items of printed material). Insome embodiments, the entities requiring the print jobs rely on theprint procurement service provider to locate suppliers on their behalf.Typically, the print procurement service provider assigns a printprocurement manager (“PPM”) to the entity and the PPM is responsible forinteracting with the print procurement system 100 (e.g., through theprint procurement web application or print procurement application) tolocate appropriate suppliers. In some embodiments, the entitiesrequiring print jobs are referred to herein as clients of the printprocurement system or as clients of the print procurement serviceprovider. In some embodiments, a PPM or some other representative of theclient interacts with the print procurement application or the printprocurement web application.

FIG. 2 is a block diagram illustrating one of the computing devices 102(e.g., computing device 102-1). In some embodiments, computing device102 is a customer device that a user uses to interact with alocally-installed print procurement application (e.g., print procurementapplication 224) and/or to interact with a print procurement webapplication (e.g., print procurement web application 222) using a webbrowser (e.g., web browser 220), in accordance with some embodiments. Insome embodiments, computing device 102 is a supplier device that a useruses to interact with a locally installed supplier portal and/or tointeract with a supplier portal web application. In some embodiments,computing device 102 is a facilitator device that a user uses to performadministration tasks such as reviewing information provided by customersand suppliers and/or managing software and/or hardware used by customersand facilitators, e.g., using a locally installed application and/or aweb application.

A computing device 102-1 typically includes one or more processingunits/cores (CPUs/GPUs) 202 for executing modules, programs, and/orinstructions stored in the memory 214 and thereby performing processingoperations; one or more network or other communications interfaces 204;the memory 214; and one or more communication buses 212 forinterconnecting these components. The communication buses 212 mayinclude circuitry that interconnects and controls communications betweensystem components.

The computing device 102-1 also includes a user interface 206 thatincludes a display device 208 and/or one or more input devices ormechanisms 210, such as a mouse, keyboard, stylus, and/or touchscreendisplay. In some embodiments in some embodiments, the inputdevice/mechanism includes a “soft” keyboard, which is displayed asneeded (e.g., in response to a user tapping a user input element thatrequires input from a keyboard, the soft keyboard is displayed) on thedisplay device 208, enabling a user to “press keys” that appear on thedisplay 208. In some embodiments, the display 208 and inputdevice/mechanism 210 comprise a touch-screen display (also called atouch-sensitive display).

In some embodiments, the computing device 102-1 has memory 214 thatincludes high-speed random access memory, such as DRAM, SRAM, DDR RAM,or other random access solid state memory devices. In some embodiments,the memory 214 includes non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid-state storage devices.Optionally, the memory 214 includes one or more storage devices remotelylocated from the CPU(s)/GPUs 202. The memory 214, or alternately thenon-volatile memory device(s) within the memory 214, comprises anon-transitory computer-readable storage medium. In some embodiments,the memory 214, or the non-transitory computer-readable storage mediumof the memory 214, stores the following programs, modules, and datastructures, or a subset thereof:

-   -   an operating system 216, which includes prodedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a communications module 218, which is used for connecting the        computing device 102-1 to computers and devices via the one or        more communication interfaces 204 (wired or wireless) and one or        more communication networks 104 (FIG. 1), such as the Internet,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   a web browser 220 (or other client-side application capable of        displaying web pages), which enables a user to communicate over        a network with remote computers, servers, and/or devices. In        some embodiments, the web browser 220 executes a print        procurement web application 222 provided by one or more print        procurement servers 106 (e.g., by receiving information about        web pages from the server 104 as needed to fulfill requests to        view information). In some embodiments, a print procurement web        application 222 is an alternative to storing a print procurement        application 224 locally on the computing device 102-1;    -   a print procurement application 224, which enables users to        interact with a print procurement system 100 (FIG. 1). In some        embodiments, the print procurement application 224 retrieves        information from one or more print job databases 108 (FIG. 1)        that are stored on one or more print procurement server(s), then        generates and displays the retrieved information in user        interfaces that allow the users to, for example, create print        jobs, identify print suppliers, select print suppliers, and        generate baseline candidates using multiple baseline        methodologies (e.g., some exemplary user interfaces are shown in        FIGS. 6A-6F and 8A-8P). In some instances, the print procurement        application 224 invokes other modules (either on the computing        device 102-1 or at one or more print procurement servers 106, to        generate baseline candidates using multiple baseline        methodologies, as described in more detail below. The print        procurement application 224 also includes one or more of the        following additional modules, or a subset or superset thereof:        -   a baseline candidate designation module 226 for designating            historical (i.e., previously fulfilled) print jobs (or items            within a print job) as baseline candidates for an            unfulfilled or newly-created print job (or item within a            print job);        -   a print job data fetching module 228 for retrieving            information from databases that are locally-stored on the            device 102-1 or that are remotely-located from the device            102-1 (e.g., from one or more print job databases 1.08 on            one or more print procurements servers 106);        -   supplier evaluation interface 230 for displaying prompts for            a user to provide supplier evaluation input and/or for            displaying information (e.g., evaluation results) about            suppliers; and/or        -   supplier rakings interface 232 for displaying information            about suppliers and/or controls for altering the way in            which information about suppliers is displayed.

In some embodiments, the memory 214, or the non-transitorycomputer-readable storage medium of the memory 214, further stores asupplier survey interface 234 that is used by suppliers to provideand/or edit information about supplier capabilities.

In some embodiments, the memory 214, or the non-transitorycomputer-readable storage medium of the memory 214, further stores asupplier survey management interface 236 that is used by a facilitatorto review information provided by suppliers using supplier surveyinterface 234.

Each of the above identified executable modules, applications, or setsof procedures may be stored in one or more of the previously-mentionedmemory devices, and corresponds to a set of instructions for performinga function described above. The above-identified modules or programs(i.e., sets of instructions) need not be implemented as separatesoftware programs, procedures, or modules, and thus various subsets ofthese modules may be combined or otherwise re-arranged in variousembodiments. In some embodiments, the memory 214 may store a subset ofthe modules and data structures identified above. Furthermore, thememory 214 may store additional modules or data structures not describedabove. In some embodiments, the programs, modules, and data structuresstored in the memory 214, or the non-transitory computer-readablestorage medium of the memory 214, provide instructions for implementingsome of the methods described below.

Although FIG. 2 shows a computing device 102-1 in accordance with someembodiments, FIG. 2 is intended more as a functional description of thevarious features that may be present rather than as a structuralschematic of the embodiments described herein. In practice, and asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated.

FIG. 3 is a block diagram illustrating a print procurement server 106-1(e.g., one or the one or more print procurement servers 106, FIG. 1), inaccordance with some embodiments. The print procurement server 106-1 mayhost one or more print job databases 108 and may provide variousexecutable applications or modules. A server 106-1 typically includesone or more processing units (CPUs/GPUs) 302, one or more networkinterfaces 304, a memory 314, and one or more communication interface(s)312 for interconnecting these components. In some embodiments, theserver 106-1 optionally includes a user interface 306, which includes adisplay device 308 and one or more input devices 310, such as a keyboardand a mouse.

The memory 314 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices, andmay include non-volatile memory, such as one or more magnetic diskstorage devices, optical disk storage devices, flash memory devices, orother non-volatile solid state storage devices. In some embodiments, thememory 314 includes one or more storage devices remotely located fromthe CPU(s)/GPUs 302. The memory 314, or alternately the non-volatilememory device(s) within the memory 314, includes a non-transitorycomputer-readable storage medium. In some embodiments, the memory 314 orthe non-transitory computer-readable storage medium of the memory 314stores the following programs, modules, and data structures, or a subsetthereof:

-   -   an operating system 316, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communications module 318, which is used for        connecting the server 106-1 to other computers via the one or        more communication interfaces 304 (wired or wireless) and one or        more communication networks 104 (FIG. 1), such as the Internet,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   a print procurement web application 222 (or information about        the print procurement application), which may be downloaded and        executed by a web browser 220 (FIG. 2) on a user's computing        device 102. In some embodiments, the print procurement web        application 222 allows the server 106-1 to serve information        related to the print procurement system 100 to one or more user        computing devices 102 (FIG. 1);    -   a baseline candidate designation module 110 for designating        historical (i.e., previously fulfilled) print jobs (or items        within a print job) as baseline candidates for an unfulfilled        print job (or item within a print job);    -   one or more print job databases 108 for storing information        about print jobs. In some embodiments, the one or more print job        databases 108 include one or more of the following additional        modules, or a subset or superset thereof:        -   information about print jobs 322, such as print            specifications associated with each line item within a            respective print job, baseline candidates associated with            the respective line item, and the like, and as discussed in            more detail below in reference to FIGS. 4A-4 B; and        -   information about baseline match tolerances 324 associated            with items in a print job, as discussed in more detail below            in reference to FIGS. 5A-5B;    -   a supplier evaluation module 326 that performs evaluation        operations, e.g., receiving user provided supplier evaluation        input, determining evaluation values, and/or transmitting        supplier evaluation information for display by a remote device;        and    -   a supplier survey module 328 that is used to:        -   generate user interfaces for receiving survey definitions            (e.g., from facilitators and/or suppliers),        -   receive and store information entered into surveys (e.g.,            from suppliers), and/or        -   evaluate information entered into surveys.

Each of the above identified executable modules, applications, or setsof procedures may be stored in one or more of the previously-mentionedmemory devices, and corresponds to a set of instructions for performinga function described above. The above-identified modules or programs(i.e., sets of instructions) need not be implemented as separatesoftware programs, procedures, or modules, and thus various subsets ofthese modules may be combined or otherwise re-arranged in variousembodiments. In some embodiments, the memory 314 may store a subset ofthe modules and data structures identified above. Furthermore, thememory 314 may store additional modules or data structures not describedabove. In some embodiments, the programs, modules, and data structuresstored in the memory 314, or the non-transitory computer-readablestorage medium of the memory 314, provide instructions for implementingsome of the methods described below.

In some embodiments, the server 106-1 and the components thereon (e.g.,print job databases 108) are implemented using a customized version ofMicrosoft Dynamics® AX on the back-end of the system.

Although FIG. 3 shows a server 106-1, FIG. 3 is intended more as afunctional description of the various features that may be presentrather than as a structural schematic of the embodiments describedherein. In practice, and as recognized by those of ordinary skill in theart, items shown separately could be combined and some items could beseparated. In addition, some of the programs, functions, procedures, ordata shown above with respect to a server 106-1 may be stored on acomputing device 102-1. In some embodiments, the functionality and/ordata may be allocated between one or more computing devices 102 and oneor more servers 106 (as explained in more detail above in reference toFIG. 1). Furthermore, one of skill in the art recognizes that FIG. 3need not represent a single physical device. In many embodiments, theserver functionality is allocated across multiple physical devices thatcomprise a server system. As used herein, references to a “server” or“print procurement server” include various groups, collections, orarrays of servers that provide the described functionality, and thephysical servers need not be physically collocated (e.g., the individualphysical devices could be spread throughout the United States orthroughout the world).

FIGS. 4A-4 B are block diagrams illustrating data structures for storinginformation about print jobs, in accordance with some embodiments. Asshown in FIG. 4A, print jobs 322 (e.g., those that are included in thedatabases 108, FIG. 3) include a collection of data structures 322,optionally implemented as a collection of tables for each particularclient of a print procurement system 100, that each store informationabout print jobs associated with the particular client of the printprocurement system 100 (e.g., print job data structure 322-1 storesinformation about print jobs for client 1 of the print procurementsystem 100 and print job data structure 322-2 stores information aboutprint jobs for client 2 of the print procurement system 100). In someembodiments, each table (e.g., table 322-1, 322-2, 322-3 . . . 322-N) inthe collection of print job data structures stores information aboutprint jobs for more than one client (e.g., print job data structure322-1 stores information about print jobs for client 1 and for client 3,because clients 1 and 3 are related corporate entities and often requestsimilar or overlapping print jobs, making storage in a single table anefficient option).

In some embodiments, determining whether to store information aboutprint jobs for more than one client or for a single client in arespective table 322 is determined when a new client begins using theprint procurement system. For example, the print procurement systemscans through the print job data structures 322 in order to determinewhether any related corporate entities, associated with a new client,are already enrolled and using the print procurement system. Inaccordance with a determination that a related corporate entity,associated with the new client, is using the print procurement system,then a respective print job data structure 322 associated with therelated corporate entity is formatted to store information about printjobs associated with the new client in addition to the related corporateentity. In some embodiments, the new client, while enrolling to use theprint procurement system, provides information about related corporateentities and this information is used to efficiently scan through theprint job data structures 322 (e.g., the scan only looks for informationabout related corporate entities matching those provided by the newclient).

In some embodiments, the clients are customers of the print procurementservice provider (discussed above in reference to FIG. 1). In someembodiments, the clients are each assigned to a print procurementmanager (“PPM”) responsible for interacting with the print procurementsystem and locating suppliers to fulfill print jobs on behalf of theirassigned clients.

In some embodiments, one or more print job data structures 322 (e.g.,print job data structure 322-1, FIG. 4B) are used for storinginformation about print jobs associated with a particular client of theprint procurement system 100. As illustrated in FIG. 4B, print job datastructure 322-1 contains data (e.g., print job ID, line item, producttype, fulfilled, and print specifications) associated with print jobsfor client 1. In some embodiments, the data is stored in individualrecords 410-1, 410-2, 410-3, 410-4, 410-5 through 410-N. In someembodiments, a header field 410-0 is used to describe each field ofinformation associated with each of the respective records. Header410-0, in some embodiments, contains a brief description of each fieldof information (e.g., each field associated with each of the records)stored within the table. As shown in FIG. 4B, header 410-0 indicatesthat each record 410-1, 410-2, 410-3, 410-4, 410-5 through 410-N storesthe following information and data structures, or a subset or supersetthereof:

-   -   a value for an “item ID” field that uniquely identifies each        line item within a particular print job;    -   a value for a “print job ID” field that uniquely identifies each        particular print job. In some embodiments, each particular print        job is associated with one or more line items;    -   a value for an “item name” field that corresponds to descriptive        text about a line item within a print job (e.g., descriptive        text provided by the client (or by a PPM working on behalf of        the client) at the time of creating a print job, such as the        text provided in a name entry field 850, FIG. 8D);    -   a value for an “item type” field that corresponds to information        about the type of product associated with the line item with        which the record is associated (e.g., information selected or        provided by a user at the time of creating a line item within a        print job (e.g., provided by selecting an option within the item        dropdown field 851 of FIG. 8D));    -   a value for a “fulfilled?” field that corresponds to information        indicating whether a line item (or print job that includes the        line item) associated with the record has been fulfilled by a        supplier (in other words the fulfilled field indicates whether        the line item is completed or is awaiting completion); and    -   a value for a “print specifications” field that includes a set        of information provided by the client (or by a PPM working on        behalf of the client) that describes the print requirements for        the particular line item (e.g., information provided in one or        more item specifications windows, such as exemplary item        specifications windows shown in FIGS. 8E-8H). In some        embodiments, the set of information in the print specifications        field includes values for each of the following print        specifications: quantity, finished size, flat size, paper        weight, paper type, print colors, coating information, basic        finishing information, complex finishing information, packing        information, delivery information, proofing information,        substrate selection information, ink information, and kitting        information. In other embodiments, the set of information in the        print specifications field includes values for only one or more        of the aforementioned print specifications.

In some embodiments, print job data structure 322-1 includes fields inaddition to the fields described above, such as a timestamp field thatidentifies when a respective line item was created and/or stored in thetable 322-1, a related line items field that includes one or more itemID values that correspond to line items that are associated with similarprint specifications, and/or a supplier assigned field that indicateswhether the particular line item has been assigned/associated with acapable supplier (i.e., a supplier with appropriate printing equipmentto fulfill the line item in accordance with the provided printspecifications that are associated with the line item).

As shown in FIG. 4B, the print job data structure 322-1 includes records410-1, 410-2, 410-3, 410-4, 410-5 through 410-N. As one example of theinformation stored in an exemplary record in the print job datastructure 322-1, record 410-1 stores a value of “1” in the item IDfield, a value of 1 in the job ID field, a value of “Spring 2000 Poster”in the item name field, a value of “Poster” in the item type field, avalue of “Yes” in the fulfilled field, and a value of “{quantity: 1000,finishedSize: 12×12, paperWeight: heavy, coatingInfo: 100# gloss}” inthe print specifications field.

Although illustrated as a particular implementation of a data structure(e.g., a table), the data structures of FIGS. 4A-4B, in someembodiments, are implemented in XML files, tables within a relationaldatabase, a document-oriented database, text files, and/or any othersuitable format for storing data.

FIGS. 5A-5B are block diagrams illustrating data structures for storinginformation about baseline match tolerances, in accordance with someembodiments. As shown in FIG. 5A, baseline match tolerance datastructures 324 include a collection of data structures 324, optionallyimplemented as a collection of tables for each particular client of theprint procurement system 100, that each store baseline match tolerancesassociated with a particular client of the print procurement system 100(e.g., baseline match tolerance data structure 324-1 stores baselinematch tolerances client 1's print jobs and baseline match tolerance datastructure 324-2 stores baseline match tolerances client 2's print jobs).In some embodiments, each table (e.g., table 324-1, 324-2, 324-3 . . .324-N) in the collection of calendar entry data structures storesbaseline match tolerances for more than one client of the printprocurement system.

In some embodiments, one or more baseline match tolerance datastructures 324 (e.g., baseline match tolerance data structure 324-1,FIG. 5B) are used for storing baseline match tolerances associated witha client of the print procurement system 100 (or for storing baselinematch tolerances associated with each print job associated with theclient). As illustrated in FIG. 5B, baseline match tolerance datastructure 324-1 contains data (e.g., item ID, job ID, exact matchtolerance, similar match tolerance, and inferred match tolerance)associated with line items and print jobs for the client. In someembodiments, the data is used to generate baseline match candidates inaccordance with the specified tolerances. For example, the similar matchtolerance field specifies acceptable deviations in print specifications(e.g., quantity can deviate by +/− 10%) that are used to locatepreviously-fulfilled line items that have print specifications thatsatisfy the similar match tolerance (e.g., quantity deviates by 5%, butall other print specifications match). In some embodiments, the data isstored in individual records 510-1, 510-2 through 510-N. In someembodiments, a header field 510-0 is used to describe each field ofinformation associated with each of the respective records. Header510-0, in some embodiments, contains a brief description of each fieldof information (e.g., each field associated with each of the records)stored within the table. As shown in FIG. 5B, header 510-0 indicatesthat each record 510-1, 510-2 through 510-N stores the followinginformation and data structures, or a subset or superset thereof:

-   -   a value for an “item ID” field that uniquely identifies each        line item within a particular print job;    -   a value for a “print job ID” field that uniquely identifies each        particular print job. As mentioned above, in some embodiments,        each particular print job is associated with one or more line        items;    -   a value for an “exact match tolerance” field that specifies a        set of deviations in print specifications for a first line item        that, if satisfied by a second line item (e.g., comparing print        specifications associated with the second line item, such as        those stored in a respective print job data structure 322, FIG.        4A, to print specifications associated with the first line item        to determine whether any deviations in print specifications for        the second line item satisfy the deviations specified in the        exact match tolerance field), allow the second line item to be        designated as an exact match baseline candidate for the first        line item. In some embodiments, the set of deviations for the        exact match tolerance field are specified by a PPM that is        responsible for locating a supplier for the first line item        (e.g., the PPM establishes the set of deviations after agreeing        with a customer requesting sourcing of the first line item about        the deviations that are acceptable for exact baseline matches);    -   a value for a “similar match tolerance” field that specifies a        set of deviations in print specifications for a first line item        that, if satisfied by a second line item (e.g., comparing print        specifications associated with the second line item, such as        those stored in a respective print job data structure 322, FIG.        4A, to print specifications associated with the first line item        to determine whether any deviations in print specifications for        the second line item satisfy the deviations specified in the        similar match tolerance field), allow the second line item to be        designated as a similar match baseline candidate for the first        line item. In some embodiments, the set of deviations for the        similar match tolerance field are specified by a PPM that is        responsible for locating a supplier for the first line item        (e.g., the PPM establishes the set of deviations after agreeing        with a customer requesting sourcing of the first line item about        the deviations that are acceptable for similar baseline        matches); and    -   a value for an “inferred match tolerance” field that specifies a        set of deviations in print specifications for a first line item        that, if satisfied by a second line item (e.g., comparing print        specifications associated with the second line item, such as        those stored in a respective print job data structure 322, FIG.        4A, to print specifications associated with the first line item        to determine whether any deviations in print specifications for        the second line item satisfy the deviations specified in the        inferred match tolerance field), allow the second line item to        be designated as an inferred match baseline candidate for the        first line item. In some embodiments, the set of deviations for        the inferred match tolerance field are specified by a PPM that        is responsible for locating a supplier for the first line item        (e.g., the PPM establishes the set of deviations after agreeing        with a customer requesting sourcing of the first line item about        the deviations that are acceptable for inferred baseline        matches).

In some embodiments, baseline match tolerance data structure 324-1includes fields in addition to the fields described above, such as a“contract ID” field that includes information identifying a contractfrom which the baseline match tolerance information was retrieved. Forexample, when a respective client agrees to employ the services providedby the print procurement service provider, the client and the serviceprovider agree to a set of terms that are embodied in a contract. Insome instances, that contract includes baseline match toleranceinformation for each print job or for each line item within each printjob.

Additional exemplary deviations in print specifications (that may beused in one of the baseline match tolerances) are shown in userinterface region 644 of FIG. 6F. As shown in FIG. 6F, the deviations inprint specifications may include a quantity difference, a paper weightchange of +/− 10#, ink difference of +/− 2 colors, one finishingdifference, page count +/− 50%, and sizing +/− 0.5 inch flat length orwidth. In some embodiments, the exact, similar, and/or inferred matchtolerances also include information about a total spec difference (alsoshown in user interface region 644 of FIG. 6F) that indicates a maximumnumber of deviations in print specifications. For example, if thesimilar match tolerance indicates a total spec difference of 1 and, eventhough a particular line item is associated with print specificationsthat satisfy each of the two deviations in print specificationsindividually, the particular line item would not be designated as asimilar baseline match candidate because the particular line itemdifferences in two print specifications which exceeds the total specdifference of 1.

As shown in FIG. 5B, the baseline match tolerance data structure 324-1includes records 510-1, 510-2 through 510-N. As one example of theinformation stored in an exemplary record in the baseline matchtolerance data structure 324-1, record 510-1 stores a value of “1” inthe item ID field, a value of “1” in the job ID field, a value of “{nodeviations}” in the exact match tolerance field, a value of “{quantity:10%}” in the similar match tolerance field, and a value of “{quantity:10%, paperWeight: 10#}” in the inferred match tolerance field.

Although illustrated as a particular implementation of a data structure(e.g., a table), the data structures of FIGS. 5A-5B, in someembodiments, are implemented in XML files, tables within a relationaldatabase, text files, and/or any other suitable format for storing data.

Turning now to FIGS. 6A-6F, schematics of a display are shown and areused to illustrate user interfaces for displaying and interacting withinformation about print jobs and for generating and displaying baselinematch candidates, in accordance with some embodiments. For ease ofreference, the user interfaces discussed below are described asdisplayed within a web browser but, in other embodiments, the userinterfaces may be displayed within a locally-installed application, suchas a mobile phone/device or desktop application (e.g., print procurementapplication 224). The user interfaces described in reference to FIGS.6A-6C are merely exemplary and additional user interfaces for printprocurement applications are also shown in FIGS. 6D-6F and in FIGS.8A-8P (as discussed in more detail below) and may be used to performsome of the functions described with respect to the user interfaces inFIGS. 6A-6C.

As shown in FIG. 6A, a web browser (e.g., web browser 220, FIG. 2) of auser computing device (e.g., user computing device 102-1) is used torender information received from a server (e.g., one or more printprocurement servers 106, FIG. 1) in order to display a print procurementweb application (e.g., print procurement web application 222, FIG. 2).In some embodiments, the print procurement web application 222 displaysinformation about print jobs that is retrieved from one or more servers106 (FIGS. 1 and 3) and the displayed information includes userinterface region 612. In some embodiments and as shown in FIG. 6A, theuser interface region 612 includes details for a particular line itemwithin a particular print job (e.g., item ID 5 of print job 4). In someembodiments, the user interface region 612 includes a preview of theitem (e.g., the displayed poster preview as shown in FIG. 6A), printspecifications associated with the item (e.g., the displayed printspecifications are retrieved from record 410-5 of print job datastructure 322-1 by querying the data structure 322-1 using item ID 5 andjob ID 4), and pricing information about the item. In some embodiments,the pricing information includes default information that is notpopulated until baseline candidates are identified and then selected forthe item (as discussed in more detail below in reference to FIGS.8M-8N). Therefore, the user interface region 612 within the printprocurement web application 222 also includes an affordance (e.g.,generate baseline candidates affordance 602) that allows the client (ora PPM acting on their behalf) to initiate a process for identifyingbaseline candidates (e.g., to initiate the method 700 discussed below).In other embodiments, the user interface region 612 has a differentvisual appearance, such as that shown in alternative user interface 612(FIG. 6D).

In some embodiments, in response to detecting user input at thedisplayed affordance 602 (e.g., in response to detecting user input 604,FIG. 6A), a process for generating baseline candidates is initiated. Insome embodiments, in response to detecting the user input 604, the printprocurement web application 222 is updated (e.g., by receiving data fromone of the print procurement servers 106) to display the user interfaceshown in FIG. 6B. In some embodiments, the user interface shown in FIG.6B displays information about baseline candidates for the line item(e.g., item ID 5 and job ID 4, “Summer 2015 Poster”) in a user interfaceregion 622. In some embodiments, the user interface region 622 includesthe poster preview discussed above and additionally includes a list ofdesignated baseline candidates for the line item. As discussed below, ahistorical line item (i.e., a historical line item is one that is a partof a completed/fulfilled print job) can be designated as either exact,baseline, inferred, or no baseline match, in accordance with acomparison of print specifications associated with the historical lineitem to print specifications associated with the line item (discussed inmore detail below in reference to FIGS. 7A-7B). In other embodiments,the user interface region 622 has a different visual appearance, such asthat shown in alternative user interface 622 (FIG. 6E).

In some embodiments, each baseline candidate within the displayed listof designated baseline candidates is selectable. In response todetecting a selection of a respective baseline candidate (e.g., inresponse to detecting user input 614), the print procurement webapplication 222 is updated (e.g., by receiving data from one of theprint procurement servers 106) to display the user interface shown inFIG. 6C. For example, the user interface shown in FIG. 6C is a baselinecalculator that allows the client (or a PPM acting on their behalf) todetermine a baseline price based on historical purchasing informationassociated with the selected baseline candidate. In some embodiments,the baseline calculator also includes a “re-designate?”button/affordance that, when selected, causes the print procurement webapplication 222 to re-designate the selected baseline candidate as adifferent type of baseline match (e.g., change an exact baseline matchto an inferred or a similar baseline match). In some embodiments, inresponse to receiving a selection of the re-designate affordance (e.g.,user input 644, FIG. 6C), the print procurement web application 222 isupdated to display a user interface element (e.g., a dialog box) thatallows for re-designating the selected baseline candidate as a differenttype of baseline match. After receiving re-designation information atthe user interface element (e.g., user input indicating that theselected baseline match should be re-designated as a similar baselinematch), the print procurement system determines whether baseline matchtolerances associated with the line item should be modified.

In some embodiments, determining whether baseline match tolerancesassociated with the line item should be modified includes analyzingprint specifications associated with the selected baseline candidate anddetermining whether additional differences in print specifications arepresent (i.e., differences between print specifications for the lineitem and those for the selected baseline candidate, distinct fromdeviations that are already included in the baseline match tolerances)and/or whether one or more deviations specified in the baseline matchtolerances should be revised. In accordance with a determination thatadditional differences in print specifications are present, the systemsuggests modifying the baseline match tolerances to account for theadditional differences. For example, the system updates web application222 to present an additional user interface that describes theadditional differences and allows the client (or a PPM acting on theirbehalf) to modify the baseline match tolerances.

Alternatively or in addition to the determination of additionaldifferences, in accordance with a determination that one or moredeviations specified in the baseline match tolerances should be revised,the system updates web application 222 to present an additional userinterface that includes information about suggested revisions to the oneor more deviations. As an example, if the selected baseline candidatehad a deviation of 6% in quantity (relative to print specificationsassociated with the line item) and thus satisfied a similar matchtolerance of up to a 10% deviation in quantity, but the client (or thePPM acting on its behalf) re-designated the selected baseline candidateas an inferred baseline match, then the system can propose modifying thesimilar baseline match threshold to have a lower percentage ofacceptable deviations in quantity. In some embodiments, the printprocurement system monitors re-designations for each client and proposesmodifications to one or more baseline match tolerances for that clientafter a threshold number of re-designations have occurred (e.g., afterthree or more). In some embodiments, the threshold number ofre-designations is different depending on the type of re-designation(e.g., exact to similar, exact to inferred, similar to inferred, etc.).For example, any re-designation that changes an exact or a similarbaseline match to an inferred baseline match may immediately trigger achange to the inferred baseline match tolerances, but the exact orsimilar baseline match tolerances remain unaffected.

In some embodiments, the baseline calculator is displayed within a userinterface region 632 on the display 208. In other embodiments, thebaseline calculator is displayed within an alternative user interface,such as alternative user interface 632 (FIG. 6F) or the user interfacesshown in FIGS. 8L-8N.

In some embodiments, the visual characteristics of the user interfacesshown in FIGS. 6A-6F include one or more visual characteristics of theuser interfaces shown in FIGS. 8A-8P.

FIG. 7 is a flowchart depicting a method 700 of generating baselinematch candidates in a print procurement system, in accordance with someembodiments. FIGS. 8A-8P are used to illustrate the methods and/orprocesses of FIGS. 7A-7B. In some embodiments, each of the userinterfaces presented in FIGS. 8A-8P are presented on a display of acomputing device (e.g., display 208 of user computing device 102-1, aspictured for the user interfaces shown in FIGS. 6A-6C). Although some ofthe examples which follow will be given with reference to inputs on atouch-sensitive display (e.g., of a mobile phone or a tablet device, inwhich a touch-sensitive surface and a display are combined), in someembodiments, the device detects inputs on a touch-sensitive surface thatis separate from the display 208 (e.g., a desktop or a laptop computingdevice).

In some embodiments, the method 700 is performed by an electronic device(e.g., computing device 102-1, FIG. 2) and/or one or more components ofthe electronic device. In some embodiments, the method 700 is governedby instructions that are stored in a non-transitory computer-readablestorage medium and that are executed by one or more processors of adevice, such as the one or more processors 202 of device 102-1 (FIG. 2).For ease of explanation, the following describes method 700 as performedby the device 102-1. In some embodiments, with reference to FIG. 2, theoperations of method 700 are performed by or use, at least in part, aweb browser (e.g., web browser 220), a print procurement web application(e.g., print procurement web application 222), a print procurementapplication (e.g., print procurement application 224), a baselinecandidate designation module (e.g., baseline candidate designationmodule 226), a print job data fetching module (e.g., print job datafetching module 228), and/or a display (e.g., display 208). Someoperations in method 700 are, optionally, combined and/or the order ofsome operations is, optionally, changed.

As described below, the method 700 provides a fast, intuitive,easy-to-use, and robust way to identify and designate baseline matchcandidates in a print procurement system. The method reduces the newlearning required of a user of a print procurement system when selectingbaseline candidates, thereby creating a more effective and efficienthuman-machine interface. Furthermore, generating baseline matchcandidates using multiple baseline methodologies in print procurementsystems allows users to create more sophisticated and accurate baselinecalculations. Further, the techniques described with regard to FIGS.7A-7B provide enhanced flexibility for identifying baseline candidatesand allow determination of a greater number of potential baselinecandidates.

Referring now to FIG. 7A, a print procurement application (e.g.,executing locally on a computing device 102 or executing through aclient-server or web-based architecture by sending information betweenthe computing device and a server 106) receives (702) a first set ofprint specifications associated with a first item in a first productcategory. In some embodiments, the first set of print specificationsincludes, e.g., quantity, finished size, flat size, paper weight, papertype, print colors, coating information, basic finishing information,complex finishing information, packing information, deliveryinformation, proofing information, substrate selection information, inkinformation, and/or kitting information. In some embodiments, productcategories include, e.g., banners, posters, and/or business cards.

In some embodiments, the first set of print specifications is receivedwhen a client (or a PPM acting on the client's behalf) enters the firstset of print specifications into a print procurement application. Insome embodiments, before entering the first set of print specifications,the PPM first creates a print job using the print procurementapplication (e.g., print procurement application 224 or printprocurement web application 222 through a web browser 220). For ease ofreference, references to the print procurement application (or simplythe application) refer to either the print procurement application orthe print procurement web application, unless explicitly notedotherwise. In some embodiments, creating a print job and entering thefirst set of print specifications includes interacting with a sequenceof user interfaces that allow the PPM to create a new job and enter theprint specifications, such as the sequence of user interfaces presentedin FIGS. 8A-8H.

For example, the PPM accesses a job list user interface (e.g., job listuser interface 880, FIG. 8A) within the print procurement applicationand selects an affordance for creating a new print job (e.g., byproviding user input 802 (e.g., a mouse click or a tap/contact using afinger) at a print job creation affordance within the job list userinterface 880). In some embodiments, the user input 802 includes a firstuser input at a menu (e.g., menu 892, FIG. 8A) and a second user inputat a new job affordance within the menu (e.g., new job affordance 893,FIG. 8A). In response to detecting the user input 802, the printprocurement application is updated to display a create new job userinterface (e.g., create new job user interface 881) on the display 208.In some embodiments, the create new job user interface 881 overlays thejob list user interface 880, such that the content within the job listuser interface 880 is visible beneath the create new job user interface881. Within the create new job user interface, the PPM enters detailsrelated to the new print job (e.g., details such as a name, a type, acustomer name, a primary PPM, a shipping date, a credit limit, and anavailable credit limit). In some embodiments, one or more of the detailsrelated to the new print job are auto-populated (i.e., populated withoutreceiving any user input) based on stored information associated withthe PPM (e.g., the primary PPM is auto-populated based on thecurrently-logged-in user and/or the credit limit and available creditare auto-populated based on information associated with the selectedcustomer).

In some embodiments, the create new job user interface 881 includes acreate new job affordance (e.g., create new job affordance 892) that,when selected, causes the print procurement application to create a newprint job with the details related to the new print job. In response todetecting user input at the create job affordance (e.g., user input804), the print procurement application creates one or more recordsassociated with the new print job and stores them in a database (e.g.,print job databases 108, discussed above in reference to FIGS. 1, 3,4A-4B, and 5A-5B). Additionally, in response to detecting the user inputat the create job affordance 892, the print procurement application isupdated to display information about the new print job. For example, insome embodiments, the print procurement application is updated todisplay a job lines listing user interface (e.g., job lines listing userinterface 882) that includes information about each line item associatedwith the new print job.

In some embodiments, the information about each line item associatedwith the new print job is displayed in a line item details region (e.g.,line item details region 894, FIG. 8C). In some embodiments, the joblines listing user interface 882 also includes a system messages region.In some embodiments, system messages 840 displays messages about systemsevents (e.g., as shown in FIG. 8C, the systems messages region 840includes an indication that a new job line has been created). In someembodiments, the job lines listing user interface 882 also includes anupdate line item affordance (e.g., update line item affordance 890) and,in response to detecting user input at the update line item affordance(e.g., user input 806), the print procurement application displays anedit line item user interface (e.g., edit line item user interface 883)that allows the PPM to modify or update information associated with oneor more line items (e.g., the banner line item that is shown within theedit line item user interface 883 in FIG. 8D) that are associated withthe currently-displayed print job (e.g., the print job created using theuser interfaces discussed above in reference to FIGS. 8A-8B). In someembodiments, selecting the updated line item affordance causes the printprocurement application to convert the information in the line itemdetails region 894 from a read-only state to an editable state. Forexample, as shown in FIG. 8D, the line item details region 894 displayseditable information associated with the selected line item (e.g.,editable line item details region 894-1). In some embodiments, the editline item user interface 883 also includes a save line item informationaffordance 871 that, when selected, causes the print procurementapplication to save any information in the editable line item detailsregion 894-1 that has been updated, modified, or added.

In some embodiments, the PPM selects a line item while on the job lineslisting user interface 882 or the edit line item user interface 883(e.g., by clicking on or tapping a select line item 870 checkbox, FIG.8C). In some embodiments, after selecting a line item in response todetecting user input at an affordance for entering print specifications(e.g., user input 808 at enter print specs 873), the print procurementapplication is updated to display user interfaces that allow for theentry of print specifications associated with the selected line item(e.g., enter print specifications user interfaces 884-1 through 884-4,as shown in FIGS. 8E-8H). In some embodiments, the enter printspecifications user interface 884-1 allows the PPM to enter printspecifications related to item size, such as flat width, flat height,finished width, finished height, finished width measurements, finishedheight measurements, and the like (FIG. 8E). In some embodiments, theenter print specifications user interface 884-2 allows the PPM to enterprint specifications related to proofing, such as an artwork file type,artwork file sent, proof type, and the like (FIG. 8F). In someembodiments, the enter print specifications user interface 884-3 allowsthe PPM to enter print specifications related to substrate selection,such as a substrate type, substrate color, banner substrate modifier,and the like (FIG. 8G). In some embodiments, the enter printspecifications user interface 884-4 allows the PPM to enter printspecifications related to the printing process, such as a print processtype, ink specifications, and the like. In some embodiments, additionalenter print specification user interfaces 884 are also available, inorder to allow the PPM to enter finishing information and kitting andpacking information (as shown in FIGS. 8E-8H). In some embodiments,selecting the save button displayed within any of the enter printspecifications user interfaces 884 causes the print procurementapplication to update a record associated with the line item, so thatthe print specifications are stored for future use and reference (e.g.,in a print specifications field of a respective record in a print jobdata structure 322, FIGS. 4A-4B).

Returning to FIG. 7A, the print procurement application retrieves (704),from a print procurement system, historical purchasing information abouta plurality of fulfilled items in the first product category. In someembodiments, the print procurement system is print procurement system100 and the print procurement application retrieves the historicalpurchasing information from user computing device 102 and from theservers 106 (or print job databases stored thereon). A fulfilled item isa line item within a print job for which printing work has beencompleted and, thus, historical purchasing information is availableabout the line item (i.e., information regarding per unit pricing forthe line item). In some embodiments, the historical purchasinginformation is retrieved by querying the one or more print job databases108 for fulfilled items in the first product category (e.g., by sendinga query that specifies fulfilled=Yes and item type=Poster).

In some embodiments, retrieving the historical purchasing informationabout the plurality of fulfilled items in the first product categoryincludes retrieving corresponding invoice data associated with theplurality of fulfilled items in the first product category. Consistentwith these embodiments, the print procurement application discardshistorical purchasing information about a respective fulfilled item ofthe plurality of fulfilled items in accordance with a determination thatthe historical purchasing information does not match the correspondinginvoice data associated with the respective fulfilled item of theplurality of fulfilled items. For example, in accordance with adetermination that historical purchasing information about therespective fulfilled item (e.g., the historical purchase informationindicates a price per unit of $10) does not match corresponding invoicedata associated with the respective fulfilled item (e.g., thecorresponding invoice data indicates a print per unit of $12). In thisway, the print procurement application is able to eliminate potentiallyerroneous or untrustworthy data and produce more accurate baselines.

In some embodiments, the print procurement application retrieves (706),from the print procurement system, an exact baseline match tolerance anda similar baseline match tolerance associated with the first item.Optionally, the print procurement application also retrieves (708) aninferred baseline match tolerance. In some embodiments, the baselinematch tolerances are retrieved from a baseline match tolerance datastructure by querying for baseline match tolerances associated with thefirst item (e.g., by querying a baseline match tolerance data structure324 for records associated with the item ID for the first item andretrieving the tolerances from the returned records, FIGS. 5A-5B providemore details regarding baseline match tolerance data structures and theinformation stored therein). In some embodiments, the exact baselinematch tolerance specifies a first plurality of deviations from the firstset of print specifications, the similar baseline match tolerancespecifies a second plurality of deviations from the first set of printspecifications, and the inferred baseline match threshold specifies athird plurality of deviations from the first set of printspecifications. Additionally, in some embodiments, the third pluralityof deviations includes more deviations than the first plurality and thesecond plurality of deviations and the second plurality of deviationsincludes more deviations than the first plurality of deviations.Exemplary deviations and baseline match thresholds are provided above inreference to FIGS. 5A-5B.

For each fulfilled item in the plurality of fulfilled items (710), theprint procurement application (i) identifies deviations in printspecifications associated with a respective fulfilled item when comparedto the first set of print specifications, (ii) in accordance with adetermination that the deviations satisfy the exact baseline matchtolerance, designates the respective fulfilled item as an exact baselinematch, and (iii) in accordance with a determination that the deviationssatisfy the similar baseline match tolerance, designates the respectivefulfilled item as a similar baseline match. As an example, the exactbaseline match tolerance specifies that quantity can deviate by up to 2%(e.g., record 510-2, FIG. 5B), the similar baseline match tolerancespecifies that quantity can deviated by up to 5% and finished size candeviate by up to 1% (e.g., record 510-2, FIG. 5B), and the deviations inprint specifications include a deviation of 3% in quantity and adeviation of 1 % in finished size. In this example, the deviations donot satisfy the exact baseline match tolerance (because there is noacceptable deviation in finished size and because the deviation inquantity exceeds 2%), but the deviations do satisfy the similar baselinematch tolerance. As such, the respective fulfilled item, after the stepsdescribed in (i), (ii), and (iii) above, would be designated as asimilar baseline match candidate.

Optionally, the print procurement application performs an additionaldetermination as to whether the deviations satisfy the inferred baselinematch tolerance and in accordance with a determination that thedeviations satisfy the inferred baseline match tolerance, the printprocurement application designates (712) the respective fulfilled itemas an inferred baseline match. In some embodiments, if the respectivefulfilled item has already been designated as an exact or similarbaseline match candidate, then the print procurement application doesnot conduct the additional determination.

In some embodiments, in accordance with a determination that thedeviations do not satisfy the exact, similar, or inferred baseline matchtolerances, the print procurement application designates (714) therespective fulfilled item as no baseline match. In some embodiments, theoperations describe above with respect to 710 are initiated in responseto user input. For example, 710 is initiated in response to userselection of a line item within a print job using select line item 870(user input 812 at select line item 870, FIG. 8J) and user selection ofa baseline request affordance 754 (user input 814 at baseline requestaffordance 754, FIG. 8K). In some embodiments, the PPM also interactswith the print procurement application to indicate that the first itemis an item for which baseline candidates should be identified, forexample, by selecting the baseline indication affordance (or checkbox)775 (user input 810 at baseline indication affordance 875, FIG. 81).

In some embodiments, the print procurement application displays (716),on a display (e.g., display 208 of user computing device 102-1, FIGS.6A-6C), each respective fulfilled item and an indication as to whethereach respective fulfilled item has been designated as an exact baselinematch, a similar baseline match, or an inferred baseline match. Anexemplary display of each respective fulfilled item is shown in FIG. 8L(e.g., baseline candidates user interface 885). In some embodiments,only those fulfilled items that have been designated as either asimilar, exact, or inferred baseline match are displayed and, thus, nobaseline match items are included in the baseline candidates userinterface 885. In some embodiments, baseline candidates user interface885 allows the PPM to quickly (A) calculate baselines or to (B)re-designate any of the baseline candidates.

As to option (A), in some embodiments, the print procurement applicationreceives (718) user input, at a location (e.g., a location on thedisplay while it is displaying the baseline candidates UI 885, FIG. 8L)that corresponds to one of the respective fulfilled items that has beendesignated as either a similar or an inferred baseline match and, basedon the user input, re-designates (720) the one respective fulfilled itemas an exact baseline match. For example, a contact with a touchscreendisplay is detected at a location on a touchscreen display thatcorresponds to a respective fulfilled item displayed in a graphical userinterface and/or an input is received while a pointer is at a locationthat corresponds to a respective fulfilled item displayed in a graphicaluser interface. In some embodiments, each of the respective fulfilleditems displayed in the baseline candidates UI 885 includes are-designate affordance (shown in FIG. 6C) that, when selected, allowsthe PPM to select a different baseline methodology for the selectedfulfilled item (i.e., change the baseline methodology fromsimilar/inferred to exact). In some embodiments, the PPM first selectsone of the respective fulfilled items that are displayed in the baselinecandidates user interface 885 (e.g., user input 816, FIG. 8L) and thenselects a re-designate element (drop-down) 878 that is shown on abaseline calculator user interface (e.g., baseline calculator userinterface 886-1, FIG. 8M) that allows for selection of a new type ofbaseline methodology. In this way, the PPM is able to flexibly configurethe types of baseline methodologies and provide user input to quicklyand easily modify a respective type of baseline methodology assigned toa respective fulfilled item, if needed or required. Additional detailsregarding re-designating baseline match candidates are provided above inreference to FIG. 6C.

As shown in FIG. 8M, baseline cost and max client sell information arenot listed until the PPM enters current bid information for the baselinecandidate. For example, and continuing the discussion of option B above,the print procurement application receives (722) a selection of one ofthe respective fulfilled items from a user interface (e.g., baselinecandidates user interface 885, FIG. 8L) displayed on the display of theelectronic device. In some embodiments, the selection is received whenthe PPM selects a baseline match candidate from baseline candidates userinterface 885 (e.g., user input 816, FIG. 8L), enters current biddetails for the baseline match candidate (e.g., enters the informationin baseline calculator user interface 886-1), and then selects acalculate button (e.g., user input 818, FIG. 8M). In response toreceiving the selection, the print procurement application proceeds todetermine a baseline price for the first item based on whether theselected fulfilled item was an exact, similar, or inferred baselinematch. In accordance with a determination that the selected fulfilleditem has been designated as an exact baseline match, the printprocurement application determines a baseline price for the first itemusing a first baseline price metric. In accordance with a determinationthat the selected fulfilled item has been designated as a similar or aninferred baseline match, the print procurement application determines abaseline price for the first item using a second baseline price metricthat is distinct from the first baseline price metric. Exemplarybaseline price metrics are provided below.

In some embodiments, the first and second price metrics are computedusing a number of equations. The equations are used to compute valuesfor the following baseline-related metrics that are shown in the tablebelow:

TABLE 1 baseline-related metrics and acronyms Max Client Sell = MCSBaseline Cost = BC Contract Margin (not percent, decimal) = CM IW TargetCost = IWTC Contract Savings (not percent, decimal) = CS Gainshare = GSIW Actual Cost = IWAC Gainshare decimal to INWK = GSINWK Gainsharedecimal to Client = GSCL Gainshare dollars to INWK = GSINWK$ Gainsharedollars to Client = GSCL$ Client previously paid record: CPPR ClientPreviously paid quantity: CPPQ Client Previously paid per unit price:CPPPUP Current Qty: CIQ Current per Unit Cost: CIPUC Current Similar BidCost: CIWCCBQ Current Cost for New Qty: CIWCNQ Similar/Inferred per Unitcost: SPUC Similar/Inferred per Unit Cost: SNPUC

As one example of the first baseline price metric, the below equationsare used to compute values for the baseline-related metrics of Table 1,in order to arrive at the baseline price:

TABLE 2 Exemplary Exact Match Baseline Equations CPPR/CPPQ = CPPPUPCPPPUP × CIQ = BC MCS = BC × (1 − CS) IWTC= MCS × (1 − CM) GS = IWTC −IWAC GSINWK = 1 − GSCL GSINWK$ = (1 − GSCL) × GS GSCL$ = GS − GSINWK$

As one example of the second baseline price metric, the below equationsare used to compute values for the baseline-related metrics of Table 1,in order to arrive at the baseline price:

TABLE 3 Exemplary Similar/Inferred Match Baseline Equations (CIWCCBQ +CS)/CPPQ = SPUC CIWCNQ/CIQ = CIPUC [(SPUC − CIPUC)/CIPUC] × CPPR = SNPUCSNPUC × CIQ = BC MCS = BC × (1 − CS) IWTC = MCS × (1 − CM) GS = IWTC −IWAC GSINWK = 1 − GSCL GSINWK$ = (1 − GSCL) × GS GSCL$ = GS − GSINWK$

Turning now to FIG. 8N, the baseline price calculator user interface isupdated to include the determined baseline price (e.g., by updating thebaseline cost and max client sell fields in baseline calculator userinterface 886-1, FIG. 8M, and then selecting the save button). In someembodiments, in response to receiving user input at a transfer button(e.g., responsible for transferring the determined baseline and baselinecost information to one or more print procurement servers 106, such asthe transfer button shown in baseline calculator UI 886-2 of FIG. 8N),the system messages 840 region is updated to indicate that the baselineinformation has been transferred (e.g., system messages 840 region ofFIG. 8O includes text indicating the information has been transferredsuccessfully). In some embodiments, the baseline cost data can also beaccessed through a baseline detail tab user interface (e.g., baselinedetail tab user interface 888, FIG. 8P).

In some embodiments, the baseline price for the first item includes(726) one or more commodities (e.g., indexed commodities) associatedwith a variable price (e.g., a price that varies based on pricing dataprovided by a third-party). In some embodiments, the print procurementapplication performs the following operations (728): (i) detects achange in the variable price (e.g., based on data received from thethird-party), (ii) determines whether the change satisfies a pricethreshold, and (iii) in accordance with a determination that the changesatisfies the price threshold, adjusts the baseline price (e.g., toaccount for the change in the variable price of the indexed commodity).

In some embodiments, the process described above with respect to FIGS.7A-7B does not locate any baseline match candidates. As such, in someembodiments, the print procurement application proceeds to conduct amulti-item baseline. A multi-item baseline allows the print procurementapplication to compare print specifications associated with more thanone item (e.g., not just a first item but a first and a second item)with groups of fulfilled items. In this way, the print procurementapplication is able to group items together in order to locate fulfilleditems that may serve as an appropriate baseline match.

It should be understood that the particular order in which theoperations in FIGS. 7A-7B have been described is merely exemplary and isnot intended to indicate that the described order is the only order inwhich the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder and/or combine theoperations described herein. Additionally, it should be noted that otherprocesses can be combined with the processes described in FIGS. 7A-7B.

In some embodiments, the print procurement application described above(and, in particular, in reference to FIGS. 7A-7B) allows for combiningmultiple print projects together into a single print job in a printprocurement system. In this way, the print procurement application isable to efficiently fulfill print jobs, reduce costs, and conserve printresources by combining similar jobs with similar print specifications.

FIG. 9 illustrates a supplier evaluation interface 230, in accordancewith some embodiments. In some embodiments, print procurementapplication 224 provides a supplier evaluation interface 230 (e.g.,displayed by a display 208 of a user device 102) that allows PPMs andother users to evaluate service provided by a supplier. In someembodiments, supplier evaluation interface 230 is displayed to a user(e.g., PPM, project manager, and/or account manager) at the time thatthe supplier invoice is sent to the user.

Supplier evaluation interface 230 includes, e.g., prompts to provide anevaluation criteria value for evaluation criteria such as the evaluationcriteria listed in Table 4 below:

TABLE 4 User Input Evaluation Criteria Min. Max. Name Type DescriptionValue Value Quote Score Whether supplier's initial 1 5 accuracy responseto the RFQ was usable or required interpretation and/or follow-up. PPMscore Range Would PPM use supplier again −1 +1 Product Score PPM'sevaluation of quality of 1 5 Quality product On-time Range Was thedelivery from the −1 +1 Delivery supplier received at the time indicatedin the supplier's delivery commitment? Invoice Range Did the supplierinvoice −1 +1 accuracy accurately reflect what was requested in thepurchase order? (E.g., a “no” response is provided when the suppliersends an invoice that includes the cost of too many produced “overs” orwhen a particular cost for a particular item or service is included inthe invoice after a previous agreement that the particular cost wouldnot be included) Value Range Did the supplier offer value −1 +1Engineering engineering (e.g., altered Services specifications in a waythat reduced cost or created better product)?

In some embodiments, the value+1 corresponds to a “yes” response, thevalue 0 corresponds to an N/A response, and the value−1 corresponds to a“no” response. For example, when a “yes” input is received at user inputcontrol 910, supplier evaluation module 326 of server 106-1 stores avalue of 1.

In some embodiments, supplier evaluation interface 230 includes inputcontrol 902 for receiving a user input value that corresponds to quoteaccuracy, input control 904 for receiving a user input value thatcorresponds to PPM score, input control 906 for receiving a user inputvalue that corresponds to on-time delivery, input control 908 forreceiving a user input value that corresponds to invoice accuracy, inputcontrol 910 for receiving a user input value that corresponds to valueengineering services, and/or input control 912 for receiving a userinput value that corresponds to product quality. While radio buttoninput controls are shown in the illustrative example of FIG. 9, it willbe recognized that drop down menus, text input fields, or other controls(including sequentially presented windows or other prompts) could beused to receive user input values for the user input evaluationcriteria.

In some embodiments, when input is received at a respective inputcontrol of input controls 902-912, a user is prompted to provide acomment. For example, when a user provides an input indicating a valueof 1 or a value of 5 at input control 912, the user is automaticallyprompted to provide a comment pertaining to the provided input.

In some embodiments, when input is received at a respective inputcontrol of input controls 902-912, a notification is sent to a remotedevice. For example, when a user provides an input indicating a value of1 or a value of 5 at input control 912, a notification is automaticallygenerated (e.g., for transmission to a facilitator user device and/or asupplier user device).

Supplier evaluation module 326 of server 106-1 receives and stores theuser input evaluation values. In some embodiments, one or more valuesprovided via supplier evaluation user interface 230 are only provided tofacilitator personnel and are not visible to other users (e.g., othercustomers and/or suppliers). For example, input provided at PPM scoreinput 904 is not visible to users other than facilitator personnel. Insome embodiments, supplier evaluation module 326 combines (e.g.,averages) one or more values provided via supplier evaluation userinterface with values provided by other users for display to PPMs andsuppliers.

In some embodiments, a user input evaluation value (e.g., productquality) for a particular invoice is determined using values provided bythe user at two different times. For example, the user is prompted toprovide a product quality score at the time that the supplier invoice isreviewed and again at a different time (e.g., three weeks from the datethat the supplier invoice is reviewed). For example, the applicationgenerates an alert to the PPM to provide a score at a date that is afterthe date that the supplier invoice is reviewed. In this way, the productquality score takes into account quality issues that are discoveredafter evaluation values are initially provided.

In some embodiments, one or more elements of supplier evaluationinterface 230 are rearranged each time the interface is displayed todiscourage quick or arbitrary data input.

In some embodiments, supplier evaluation module 326 determines one ormore system-generated evaluation criteria values for evaluation criteriasuch as the evaluation criteria listed in Table 5 below:

TABLE 5 System-generated Evaluation Criteria Min. Max. Name TypeDescription Value Value Bid response Percentage Percent of time supplier0 100% responds to a bid (including no bid response) Quote Time in Quoteturn speed (hours)— 0  336 hrs. response hours outbound to inbound hours(14 days) speed measurement Invoice Time in Speed with which invoice 08765 hrs. speed hours provided from supplier to hours (1 year) invoicerecipient (e.g., difference between shipment date and invoice date)Portal Percentage Percent of supplier invoices 0% 100% invoicingsubmitted through supplier portal Low bidder Percentage Percent ofquotes where the 0% 100% supplier offered the lowest bid WinningPercentage Win ratio (e.g., percentage 0% 100% quote of quotes thatbecome fulfilled jobs Award trend Calculated Number jobs awarded in 0Infinity number past 3, 6, and/or 12 months range Profile PercentageCompleteness of supplier 0% 100% completeness profile (e.g., number ofprofile fields completed). Unsolicited Percentage Percent of jobssourced 0% 100% bidding through portal—i.e., unsolicited bids Job StatusPercentage Number of times supplier 0% 100% Updates updated divided bynumber of times supplier was required to update

Minimum and maximum values provided in the system-generated evaluationcriteria table and the user input evaluation criteria table areexamples. It will be recognized that other values and rating systems maybe used for the various evaluation criteria.

In some embodiments, system-generated evaluation criteria values (and/orvalues determined based on representative system-generated evaluationvalues, such as on-time delivery (see FIG. 10)) are determined when aPPM submits a request for a quote, when a supplier submits a response toa request for a quote, when a supplier ships a product, when a suppliertransmits an invoice, and/or bidding phase and/or when a supplierrankings interface 232 is displayed.

FIG. 10 illustrates a supplier rankings interface 232, in accordancewith some embodiments. In some embodiments, print procurementapplication 224 provides a supplier rankings interface 232 (e.g.,displayed by a display 208 of a user device 102) that indicatesinformation about suppliers, such as evaluation values provided by usersand/or determined by a system for suppliers.

In some embodiments, supplier evaluation module 326 calculates scoresfor each supplier in the system that has bid on and/or produced aproduct that is included in a product order. In some embodiments, asupplier rakings interface 232 includes, for one more products in aproduct order, a list of all or fewer than all of the suppliers of theproduct. In some embodiments, combined and/or representative values forone or more evaluation criteria are provided for each listed supplier ofthe product. In some embodiments, the suppliers are listed in accordancewith a ranking order determined using the one or more evaluationcriteria. In some embodiments, suppliers who have not bid on or produceda product in the product order are listed with the indication “NoActivity.”

For example, supplier rankings interface 232 displays evaluation valuesdetermined for product quality, on-time delivery, invoice accuracy,value engineering, timeliness, and pricing and win rates, for suppliersSupplier A, Supplier B, and Supplier C. In some embodiments, supplierrankings interface 232 includes one or more sort controls, such asproduct quality sort control 1002, on-time delivery sort control 1004,invoice accuracy sort control 1006, value engineering sort control 1008,timeliness sort control 1010, and/or pricing and win rates sort control1012. When input is received at a respective sort control of sortcontrols 1002-1012, suppliers are sorted according to the evaluationcriterion that corresponds to the sort control. In some embodiments,subsequent inputs received at a sort control toggle the sort direction(e.g., between ascending order and descending order). For example, inresponse to an input received at on-time delivery sort control 1004, thesuppliers are sorted in descending order in accordance with “on-timedelivery” values determined for each supplier. In response to asubsequent input received at on-time delivery sort control 1004, thesuppliers would be sorted in ascending order. In some embodiments, asort direction indicator 1014 indicates a sort direction (e.g., downwardpointing icon for descending sort order and upward pointing icon forascending sort order).

In some embodiments, a view comments control 1016 is provided to allow auser to view comments pertaining to an evaluation criterion. Forexample, in response to an input received at a view comments control1016 for product quality, supplier rankings interface 232 displays oneor more comments provided by PPMs pertaining to product quality. Forexample, the displayed comments associated with a combined productquality score for a supplier include comments corresponding to threerecently provided product quality scores of “5” and commentscorresponding to three recently provided product quality scores of “1.”

In some embodiments, a timeliness score is determined using an averagequote response speed for quotes provided by a supplier and an averageinvoice speed of quotes provided by the supplier. For example, a firstweight is applied to the average quote response speed and a secondweight is applied to the average invoice speed, and the weighted valuesare averaged. In some embodiments, the first weight and the secondweight are equal.

In some embodiments, a pricing and win rates score is determined usingan average “low bidder” percentage value for quotes provided by asupplier and an average “winning quote” percentage value for quotesprovided by the supplier. For example, a first weight is applied to theaverage low bidder percentage and a second weight is applied to theaverage winning quote percentage, and the weighted values are averaged.In some embodiments, the first weight and the second weight are equal.

In some embodiments, information provided for a supplier in supplierrankings interface 232 includes a notes field indicating supplierinformation (e.g., a supplier review). For example, the supplierinformation is provided by PPMs and/or a facilitator.

In some embodiments, supplier rankings interface 232 includes one ormore controls for posting a reaction to a supplier, a review, and/or acomment provided for a supplier (e.g., input control 1018 to “like”Supplier B). In some embodiments, a PPM score is determined using thereactions to a supplier and/or comments for a supplier.

In some embodiments, user input to “follow” a supplier is detected,e.g., when a user input is received at a location corresponding to afollow control (e.g., follow control 1020 to follow Supplier B displayedin supplier rankings interface 232). When a user is following asupplier, the user is notified when new comments and/or reviews areposted for the supplier.

Supplier evaluation module 326 of server 106-1 receives user providedsupplier evaluation input, determines evaluation values, and transmitssupplier evaluation information for display by a remote device.

FIGS. 11A-11B are a flowchart 1100 representation of a method ofdisplaying supplier evaluation values, in accordance with someembodiments.

In some embodiments, the method 1100 is performed by an electronicdevice (e.g., server 106-1, FIG. 3) and/or one or more components of theelectronic device. In some embodiments, the method 1100 is governed byinstructions that are stored in a non-transitory computer-readablestorage medium and that are executed by one or more processors of adevice, such as the one or more processors 302 of device 106-1 (FIG. 3).For ease of explanation, the following describes method 1100 asperformed by the device 106-1. In some embodiments, one or moreoperations are performed by a user device 102. Some operations in method1100 are, optionally, combined and/or the order of some operations is,optionally, changed.

As described below, the method 1100 uses user input evaluationinformation and automatically generated system-generated evaluationinformation to provide information that customers can use to compare andselect suppliers. The approach described with regard to FIGS. 11A-11Bleverages information that is automatically generated as a printprocurement system is used, such as time elapsed between user actions,to generate information to be displayed in a user interface.

The device receives (1102) a print order. For example, a print ordergenerated by print procurement application 224 is transmitted from userdevice 102 and received by device 106.

In response to receiving the print order, the device (1104) analyzes asupplier data set to determine a set of potential suppliers for theprint order. In some embodiments, a supplier data set is, e.g., anarray, database (e.g., print job database 108), portion of a database,or other data structure that includes information for multiplesuppliers, including identifying information for a supplier and/orinformation indicating one or more products items that a supplier iscapable of supplying. In some embodiments, a supplier data set is storedby print procurement server 106. In some embodiments, at least a portionof the data in a supplier data set is provided by a supplier via asupplier survey interface 234. In some embodiments, at least a portionof the data in a supplier data set is input by a user via supplierevaluation interface 230. In some embodiments, a supplier data setincludes user input evaluation criteria for a supplier and/or for aproduct supplied by the supplier. In some embodiments, supplier data setincludes system-generated evaluation criteria for a supplier and/or fora product supplied by the supplier. In some embodiments, determining aset of potential suppliers includes determining a set of suppliers thatare capable of supplying at least one order item of the print order. Forexample, supplier survey input received via a supplier survey interface234 is analyzed to determine whether a supplier satisfies printspecifications for the order item.

In response to receiving the print order, for each respective supplierin the set of potential suppliers for the print order, the devicedetermines (1104), using one or more system-generated evaluationcriteria values for the respective supplier (e.g., as described withregard to Table 5), at least one representative system-generatedevaluation criterion value for the respective supplier. In someembodiments, the at least one representative system-generated evaluationcriterion value is based on one or more system-generated evaluationcriteria values (e.g., values provided by one or more suppliers for aparticular evaluation criterion or for multiple evaluation criteria)stored in a supplier data set for a supplier and/or a product producedby the supplier. For example, the at least one representativesystem-generated evaluation criterion value is an award trend valuedetermined based on a number of jobs awarded in the three monthspreceding the current date. In some embodiments, supplier evaluationmodule 326 stores determined system generated evaluation evaluationinformation (e.g., to memory 214 and/or database 108) and determines arepresentative system-generated evaluation criterion value using thestored system-generated evaluation information.

In response to receiving the print order, for each respective supplierin the set of potential suppliers for the print order, the devicedetermines (1104), using one or more user input evaluation criteriavalues for the respective supplier (e.g., as described with regard toTable 4), at least one combined user input evaluation criterion valuefor the respective supplier. In some embodiments, the at least onecombined user input evaluation criterion value is based on one or moreuser input evaluation criteria values (e.g., e.g., values provided byone or more suppliers for a particular evaluation criterion or formultiple evaluation criteria) stored in a supplier data set for asupplier and/or a product produced by the supplier. For example, the atleast one combined user input evaluation criterion value is an averageof every invoice accuracy value provided (e.g., via supplier evaluationinterface 230) by PPMs for a particular supplier. In some embodiments,supplier evaluation module 326 stores received user input evaluationinformation (e.g., to memory 214 and/or database 108) and determines acombined user input evaluation criterion value using the stored userinput evaluation information.

In some embodiments, the one or more input evaluation criteria valuesare determined (1106) based on user input data received from a device(e.g., user device 102-1) that is remote from the electronic device(e.g., server 106-1). For example, user input data is received at userdevice 102 via supplier evaluation interface 230 and transmitted fromuser device 102 to server 106.

In some embodiments, the one or more system-generated evaluationcriteria values are automatically determined (1108) based on operationsof an application. Operations of an application include operations suchas a storage event and/or a triggering event (such as print orderreceived, quote response provided, shipping date entry occurs, invoicetransmission occurs, bid determined to be low bid, bid accepted, and/ora supplier profile is completed). For example, a first time value isstored when a request for a quote is received by server 106 from a firstuser device 102-1 (e.g., from a PPM), and a second time value is storedwhen a response to the request is received by server 106 from a seconduser device 102-2 (e.g., from a supplier). In another example, a firsttime value is stored when a shipping date entry occurs, and a secondtime entry is stored when invoice transmission from a supplier to a PPMoccurs. When a triggering event occurs, a time duration between thefirst time and the second time is automatically determined. In anotherexample of an operation of an application, when a bid provided by asupplier is determined to be a low bid among multiple bids provided bymultiple suppliers, a low bid counter for the supplier is automaticallyincremented. In a further example of an operation of an application,when a bid provided by a supplier is determined to be a winning bid(e.g., a PPM indicates acceptance of the bid via print procurementapplication 224), a winning bid counter for the supplier isautomatically incremented.

In some embodiments, the at least one representative system-generatedevaluation criterion value for the respective supplier includes (1110) atimeliness value that is determined using: one or more quote responsetime duration values for the respective supplier, and one or moreinvoice speed duration values for the respective supplier. For example,an average quote response time duration value and an average invoicespeed duration value are weighted (e.g., equally weighted) and averaged.

In some embodiments, the at least one representative system-generatedevaluation criterion value for the respective supplier includes (1112) apricing and win rates score that is determined using: a first ratio ofwinning quotes provided by the respective supplier to total quotesprovided by the respective supplier, and a second ratio of low bidquotes provided by the respective supplier to total quotes provided bythe respective supplier. For example, the first ratio and the secondratio are weighted (e.g., equally weighted) and averaged.

In some embodiments, the device displays (1114) supplier information fora subset of the set of potential suppliers for the print order (e.g.,every supplier in the set of potential suppliers or less than all of thesuppliers in the set of potential suppliers), wherein displayed supplierinformation includes: (1) identifying information for a respectivesupplier from the subset of the set of potential suppliers for the printorder, (2) a representative system-generated evaluation criterion valuefor the respective supplier from the subset of the set of potentialsuppliers for the print order, and (3) a combined user input evaluationcriterion value for the respective supplier from the subset of the setof potential suppliers for the print order. For example, as describedwith regard to FIG. 10, supplier rankings interface 232 displayssupplier information for suppliers Supplier A, Supplier B, and SupplierC. The information in FIG. 10 includes identifying information for arespective supplier (e.g., “Supplier A”), a representativesystem-generated evaluation criterion value for the respective supplier(e.g., a timeliness value of−0.59), and a combined user input evaluationcriterion value for the respective supplier (e.g., a product qualityvalue of 4.2).

In some embodiments, for each respective supplier in the set ofpotential suppliers for the print order, the device determines (1116) aranking for a respective supplier using at least one of a representativesystem-generated evaluation criterion value for the respective supplieror a combined user input evaluation criterion value for the respectivesupplier. For example, as described with regard to FIG. 10, a rankingfor suppliers Supplier A, Supplier B, and Supplier C is determined basedon a representative on-time delivery value (determined based on acombination of a system-generated quote response speed evaluationcriterion and a system-generated invoice speed evaluation criterion) inresponse to input received at control 1004. If a user were to provideinput at invoice accuracy control 1006, Supplier A, Supplier B, andSupplier C would be re-ranked according to combined user-input invoiceaccuracy evaluation for the supplier. In some embodiments, the number ofsuppliers for which rankings are determined is greater than the numberof suppliers displayed in supplier rankings interface 232. In this way,the suppliers to be displayed in supplier rankings interface 232 can belimited to a predetermined number of highest ranked suppliers.

In some embodiments, the subset of the set of potential suppliers forthe print order includes (1118) a predefined number of highest rankedsuppliers (e.g., three suppliers, as shown in FIG. 10) based on thedetermined rankings for each respective supplier in the set of potentialsuppliers for the print order.

In some embodiments, the subset of the set of potential suppliers forthe print order is displayed (1120) in an order that is based on thedetermined rankings for each respective supplier in the set of potentialsuppliers for the print order. For example, in FIG. 10, suppliersSupplier A, Supplier B, and Supplier C are displayed in descending orderbased on on-time delivery value rankings.

It should be understood that the particular order in which theoperations in FIGS. 11A-11B have been described is merely exemplary andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder and/or combine theoperations described herein. Additionally, it should be noted that otherprocesses can be combined with the processes described in FIGS. 11A-11B.

In some embodiments, a supplier portal includes one or more graphicaluser interfaces to display information to suppliers and receive inputfrom suppliers. Suppliers use the supplier portal to, e.g., providebusiness information, search requests for quotes, respond to requestsfor quotes, view the statuses of quotes, receive notifications (e.g.,notifications about new requests for quotes and/or accepted bids),provide status updates to PPMs, and/or perform invoicing.

In some embodiments, a supplier portal includes a supplier surveyinterface 234 that can be used by suppliers to indicate and/or editinformation regarding the supplier's equipment, capabilities, and/orproduction capacity. For example, the interface is used to indicateproduct specifications (e.g., poster size) that can be fulfilled by thesupplier. In some embodiments, when an edit is made to equipment orcapabilities for a supplier, an automatic notification is generated(e.g., by a supplier survey module 328 of server 106) for transmissionto a facilitator and/or any PPM who is following the supplier (e.g., asdescribed above with regard to follow control 1020 of FIG. 10).

Supplier survey interface 234 includes one or more fields for receivinginformation entered by suppliers. The information includes, e.g.,equipment information (e.g., for one or more types of productionequipment owned by the supplier: an equipment identifier, an equipmentcount, and/or information indicating one or more locations of theequipment), information indicating types of products that the supplieris able to produce (e.g., posters, banners, brochures, calendars,business cards, etc.), types of substrates used by a supplier (e.g.,vinyl, paper, etc.), and/or dimensions of products that the supplier isable to produce (e.g., for one or more types of equipment, one or moreproduct types, and/or one or more substrate types: a height, width,minimum quantity and/or maximum quantity).

In some embodiments, a supplier survey design tool (e.g., suppliersurvey management interface 236) allows a system user (e.g., a PPM orother customer personnel, and/or personnel from a facilitating entity)to configure and/or manage the fields that appear in the supplier surveyinterface 234. In some embodiments, a supplier is enabled to add fieldsto the supplier survey interface 234, and supplier survey managementinterface 236 allows configuration of the fields that are available fora supplier to add to the interface.

In some embodiments, suppliers are enabled to upload certificationsand/or classifications (e.g., minority owned business, foreststewardship council, international standards organization, etc.) viasupplier survey interface 234. In some embodiments, a scan of a currentcertification document is required to be entered via the interface 234for verification of authenticity.

In some embodiments, supplier survey interface 234 displays suppliercertifications and/or provides controls for a supplier to editcertification information.

In some embodiments, a facilitating entity stores customer preferencesand/or required certifications (e.g. at print procurement server(s) 106)to allow suppliers to be matched with customers. In some embodiments,supplier information is displayed to customers in accordance withcustomer preferences and/or required certifications. For example, if aPPM indicates a preference for printing with a particular type ofequipment, the information is used to filter suppliers in accordancewith equipment information provided via supplier survey interface 234.

In some embodiments, a facilitating entity uses a supplier surveymanagement interface 236 to perform supplier survey approval beforeresponses received from a supplier via supplier survey interface 234 areavailable for viewing by a PPM. For example, when a supplier completes asurvey, a report is generated (e.g., in tabular form) that indicates theinformation provided by a supplier. Typically, the report is generatedin an easily readable format that allows a facilitator to quickly reviewthe entered information.

The terminology used in the description of the invention herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of the invention. As used in the description ofthe invention and the appended claims, the singular forms “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will also be understood that theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, steps, operations, elements, and/or components, but donot preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups thereof.

It will also be understood that, although the terms “first,” “second,”etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These tern's are only used todistinguish one element from another. For example, a first region couldbe termed a second region, and, similarly, a second region could betermed a first region, without changing the meaning of the description,so long as all occurrences of the “first region” are renamedconsistently and all occurrences of the “second region” are renamedconsistently. The first region and the second region are both regions,but they are not the same region.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method of displaying supplier evaluationvalues, comprising: at an electronic device with one or more processorsand memory: receiving a print order; in response to receiving the printorder: analyzing a supplier data set to determine a set of potentialsuppliers for the print order, and for each respective supplier in theset of potential suppliers for the print order: determining, using oneor more system-generated evaluation criteria values for the respectivesupplier, at least one representative system-generated evaluationcriterion value for the respective supplier, and determining, using oneor more user input evaluation criteria values for the respectivesupplier, at least one combined user input evaluation criterion valuefor the respective supplier; and displaying supplier information for asubset of the set of potential suppliers for the print order, whereindisplayed supplier information includes identifying information, arepresentative system-generated evaluation criterion value, and acombined user input evaluation criterion value for a respective supplierfrom the subset of the set of potential suppliers for the print order.2. The method of claim 1, wherein the one or more input evaluationcriteria values are determined based on user input data received from adevice that is remote from the electronic device.
 3. The method of claim1, wherein the one or more system-generated evaluation criteria valuesare automatically determined based on operations of an application. 4.The method of claim 1, wherein the at least one representativesystem-generated evaluation criterion value for the respective supplierincludes a timeliness value that is determined using: one or more quoteresponse time duration values for the respective supplier, and one ormore invoice speed duration values for the respective supplier.
 5. Themethod of claim 1, wherein the at least one representativesystem-generated evaluation criterion value for the respective supplierincludes a pricing and win rates score that is determined using: a firstratio of winning quotes provided by the respective supplier to totalquotes provided by the respective supplier, and a second ratio of lowbid quotes provided by the respective supplier to total quotes providedby the respective supplier.
 6. The method of claim 1, including, foreach respective supplier in the set of potential suppliers for the printorder, determining a ranking for a respective supplier using at leastone of a representative system-generated evaluation criterion value forthe respective supplier or a combined user input evaluation criterionvalue for the respective supplier.
 7. The method of claim 6, wherein:the subset of the set of potential suppliers for the print orderincludes a predefined number of highest ranked suppliers based on thedetermined rankings for each respective supplier in the set of potentialsuppliers for the print order.
 8. The method of claim 6, wherein: thesubset of the set of potential suppliers for the print order isdisplayed in an order that is based on the determined rankings for eachrespective supplier in the set of potential suppliers for the printorder.
 9. An electronic device, comprising: a display; memory; one ormore processors; and memory storing one or more programs, which whenexecuted by the one or more processors cause the electronic device to:receive a print order; in response to receiving the print order: analyzea supplier data set to determine a set of potential suppliers for theprint order, and for each respective supplier in the set of potentialsuppliers for the print order: determine, using one or moresystem-generated evaluation criteria values for the respective supplier,at least one representative system-generated evaluation criterion valuefor the respective supplier, and determine, using one or more user inputevaluation criteria values for the respective supplier, at least onecombined user input evaluation criterion value for the respectivesupplier; and display supplier information for a subset of the set ofpotential suppliers for the print order, wherein displayed supplierinformation includes identifying information, a representativesystem-generated evaluation criterion value, and a combined user inputevaluation criterion value for a respective supplier from the subset ofthe set of potential suppliers for the print order.
 10. The electronicdevice of claim 9, wherein the one or more input evaluation criteriavalues are determined based on user input data received from a devicethat is remote from the electronic device.
 11. The electronic device ofclaim 9, wherein the one or more system-generated evaluation criteriavalues are automatically determined based on operations of anapplication.
 12. The electronic device of claim 9, wherein the at leastone representative system-generated evaluation criterion value for therespective supplier includes a timeliness value that is determinedusing: one or more quote response time duration values for therespective supplier, and one or more invoice speed duration values forthe respective supplier.
 13. The electronic device of claim 9, whereinthe at least one representative system-generated evaluation criterionvalue for the respective supplier includes a pricing and win rates scorethat is determined using: a first ratio of winning quotes provided bythe respective supplier to total quotes provided by the respectivesupplier, and a second ratio of low bid quotes provided by therespective supplier to total quotes provided by the respective supplier.14. The electronic device of claim 9, wherein the one or more programscause the electronic device to, for each respective supplier in the setof potential suppliers for the print order, determine a ranking for arespective supplier using at least one of a representativesystem-generated evaluation criterion value for the respective supplieror a combined user input evaluation criterion value for the respectivesupplier.
 15. The electronic device of claim 14, wherein: the subset ofthe set of potential suppliers for the print order includes a predefinednumber of highest ranked suppliers based on the determined rankings foreach respective supplier in the set of potential suppliers for the printorder.
 16. The electronic device of claim 14, wherein: the subset of theset of potential suppliers for the print order is displayed in an orderthat is based on the determined rankings for each respective supplier inthe set of potential suppliers for the print order.
 17. A non-transitorycomputer-readable storage medium storing executable instructions that,when executed by an electronic device, cause the electronic device to:receive a print order; in response to receiving the print order: analyzea supplier data set to determine a set of potential suppliers for theprint order, and for each respective supplier in the set of potentialsuppliers for the print order: determine, using one or moresystem-generated evaluation criteria values for the respective supplier,at least one representative system-generated evaluation criterion valuefor the respective supplier, and determine, using one or more user inputevaluation criteria values for the respective supplier, at least onecombined user input evaluation criterion value for the respectivesupplier; and display supplier information for a subset of the set ofpotential suppliers for the print order, wherein displayed supplierinformation includes identifying information, a representativesystem-generated evaluation criterion value, and a combined user inputevaluation criterion value for a respective supplier from the subset ofthe set of potential suppliers for the print order.
 18. Thecomputer-readable storage medium of claim 17, wherein the one or moreinput evaluation criteria values are determined based on user input datareceived from a device that is remote from the electronic device. 19.The computer-readable storage medium of claim 17, wherein the one ormore system-generated evaluation criteria values are automaticallydetermined based on operations of an application.
 20. Thecomputer-readable storage medium of claim 17, wherein the at least onerepresentative system-generated evaluation criterion value for therespective supplier includes a timeliness value that is determinedusing: one or more quote response time duration values for therespective supplier, and one or more invoice speed duration values forthe respective supplier.
 21. The computer-readable storage medium ofclaim 17, wherein the at least one representative system-generatedevaluation criterion value for the respective supplier includes apricing and win rates score that is determined using: a first ratio ofwinning quotes provided by the respective supplier to total quotesprovided by the respective supplier, and a second ratio of low bidquotes provided by the respective supplier to total quotes provided bythe respective supplier.
 22. The computer-readable storage medium ofclaim 17, wherein the one or more programs cause the electronic deviceto, for each respective supplier in the set of potential suppliers forthe print order, determine a ranking for a respective supplier using atleast one of a representative system-generated evaluation criterionvalue for the respective supplier or a combined user input evaluationcriterion value for the respective supplier.
 23. The computer-readablestorage medium of claim 22, wherein: the subset of the set of potentialsuppliers for the print order includes a predefined number of highestranked suppliers based on the determined rankings for each respectivesupplier in the set of potential suppliers for the print order.
 24. Thecomputer-readable storage medium of claim 22, wherein: the subset of theset of potential suppliers for the print order is displayed in an orderthat is based on the determined rankings for each respective supplier inthe set of potential suppliers for the print order.